Recent works (e.g., (Li \& Arora, 2020)) suggest that the use of popular normalization schemes (including Batch Normalization) in today's deep learning can move it far from a traditional optimization viewpoint, e.g., use of exponentially increasing learning rates. The current paper highlights other ways in which behavior of normalized nets departs from traditional viewpoints, and then initiates a formal framework for studying their mathematics via suitable adaptation of the conventional framework namely, modeling SGD-induced training trajectory via a suitable stochastic differential equation (SDE) with a noise term that captures gradient noise. This yields: (a) A new \textquotedblleft intrinsic learning rate\textquotedblright\ parameter that is the product of the normal learning rate $\eta$ and weight decay factor $\lambda$. Analysis of the SDE shows how the effective speed of learning varies and equilibrates over time under the control of intrinsic LR.

Weaknesses: Among several, your paper makes two concrete predictions: 1. When dropping learning rate by 10, the intrinsic learning rate drops by 10 immediately (this is obvious), but it eventually converges to sqrt(10) 2. Reaching equilibrium takes O(1/\lambda_e) steps. I'd like to see experiments measuring and verifying them, or if your results are already in the paper, have them be more prominent, and linked to where these predictions are discussed. For example, I'd like to see a plot that plots 1/\lambda_e vs "step to convergence", which should be linear if your prediction is correct. Other questions I have 1.

Thank you for submitting your work to NeurIPS. All four reviewers were enthusiastic about the paper, and I am happy to accept it. In the final revision, please address reviewers' feedback. Especially, please make sure to address the reviewers' 2 remark "authors argue that their results indicate that large learning rates do not generalize well, but a better presentation would be to say that they show that large effective learning rates generalize well.". Indeed, it is somewhat a strawman argument to say that other researchers claim that small LR never generalize.

Recent works (e.g., (Li \& Arora, 2020)) suggest that the use of popular normalization schemes (including Batch Normalization) in today's deep learning can move it far from a traditional optimization viewpoint, e.g., use of exponentially increasing learning rates. The current paper highlights other ways in which behavior of normalized nets departs from traditional viewpoints, and then initiates a formal framework for studying their mathematics via suitable adaptation of the conventional framework namely, modeling SGD-induced training trajectory via a suitable stochastic differential equation (SDE) with a noise term that captures gradient noise. This yields: (a) A new \textquotedblleft intrinsic learning rate\textquotedblright\ parameter that is the product of the normal learning rate \eta and weight decay factor \lambda . Analysis of the SDE shows how the effective speed of learning varies and equilibrates over time under the control of intrinsic LR. We name it the \emph{Fast Equilibrium Conjecture} and suggest it holds the key to why Batch Normalization is effective.

You may remember our previous blog post showing that it is possible to do state-of-the-art deep learning with learning rate that increases exponentially during training. It was meant to be a dramatic illustration that what we learned in optimization classes and books isn't always a good fit for modern deep learning, specifically, normalized nets, which is our term for nets that use any one of popular normalization schemes,e.g. Today's post (based upon our paper with Kaifeng Lyu at NeurIPS20) identifies other surprising incompatibilities between normalized nets and traditional analyses. We hope this will change the way you teach and think about deep learning! Before diving into the results, we recall that normalized nets are typically trained with weight decay (aka $\ell_2$ regularization).